Machine Versus Human Clustering of Concepts Across Documents

An automated method for clustering terms/concepts from a set of documents on the same topic was developed for the purpose of multidocument summarization. The clustering method makes use of a combination of lexical overlap between multiword terms, syntactic constraints and semantic consideration based on a manually constructed taxonomy to generate hierarchically organized clusters of terms. This study evaluates the machine-generated clusters by calculating the proportion of overlap with two sets of humangenerated clusters for 15 topics. It was found that the overlap between machine-generated clusters and individual human-generated clusters are higher than that between two human-generated clusters. A qualitative analysis of the human clustering found that clusters formed are either semantic-conceptual based or lexical based (similar to machine clustering). The semantic-conceptual based clusters that were formed tended to be different for different human coders. This has raised questions about whether machine-generated clustering can be evaluated by comparing with human clustering. Introduction This paper reports a study of machine and human clustering of concepts across documents. The study is carried out in the context of multi-document summarization research—to develop an automatic method to summarize a set of related documents on a particular topic. Multi-document summarization involves identifying common information found in multiple documents, relations between the pieces of information, as well as unique information found in individual documents. Our approach to multidocument summarization focuses on extracting important terms/concepts in the documents and identifying the relations specified between the concepts. Thus an important step in the summarization method involves clustering similar or related concepts. We adopt a method for clustering concepts using a “global taxonomy” developed manually based on a corpus, in combination with a “local taxonomy” or hierarchical structure of terms constructed from the set of related documents to be summarized. In this paper, we report the results of an evaluation that compares the clusters generated by the automatic method with clusters constructed by human coders. The objectives are to find out: 1. How “good” the machine-generated clusters are compared to human-generated clusters 2. The characteristics of human-generated clusters. Clustering and categorization are a fundamental human behavior, and though there have been many studies of human categorization in the field of cognitive psychology, they have focused on categorization of common objects and concepts. We have not found any work by information science researchers on human clustering of terms and concepts taken from documents. The kind of human clustering research that is closest to ours is the card sorting studies sometimes carried out to develop menu hierarchies and taxonomies for organizing Web sites and information system interfaces. Bar-Ilan and Belous (2007) studied how children organized subject categories taken from Web